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ABSTRACT 

School psychologists spend between 40-60X of their 
time in assessment activities. Methods for improving the diagnostic 
process are addressed in this paper* Techniques mentioned to improve 
the process include the actuarial method, the multitrait multimethod 
approach to mul tif actored assessment, and direct instruction in 
reasoning and decision making* In addition, the changing role of the 
school psychologist in relation to assessment is discussed* The role 
traditional assessment still has in a school psychology practice is 
presented, suggesting that there is a need to refine and improve that 
process rather than discard it* The claim is made that the key to 
keeping school psychologists in the school setting is the expansion 
of their training in order to provide more varied services to the 
entire school community* (Contains 38 references*) (TS) 



Vc Vf Vc V? it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it if it it it it it it ?? it *c it * ? 't i c loc * f i of >t 

’' Reproductions supplied by EDRS are the best that can be made 

* , . from the original document. 

y. y. y. y. y. y. y. y. y. y. y. y. y. y. y. y. y. ;j yc y. y. y- y. ,'c yc yc yc yc * yc * it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it 




D27 1 3 6 



TECHNIQUES FOR IMPROVING THE DIAGNOSTIC PROCESS 



BY 



JOSEPH R. DAVIDOW 

OCEAN CITY N.J. SCHOOL DISTRICT 
OCEAN CITY, N.J. 08226 



PRESENTED AT THE ANNUAL MEETING OF THE 
NATIONAL ASSOCIATION OF SCHOOL PSYCHOLOGISTS (NASP) 
MARCH, 1996 ATLANTA, GA . 



BEST COPY AVAILABLE 




! } ( I 1) i'U I 'fU ■: I' i 

mi:. MAI I ^^<Al 
HAS CM N f .HANU H fv, 



mi I IJIM A)I( )NAl HI '.OHHi .f 
Tjl i .'HMAIU ‘N ( 1 NM H <\ Hlf i 



I’ hu'Am'.'i SI .1} ( i.i'! AH' *s 

I h'H AllONAL HI S(*i)H( I S l?jlOHHAT K 
( I Ml H (I I'K ‘ 

L) ! *" ■ 'i '< iiMici'l . t I'l't. * »l .I*' 

'I 'I.,. 

... -il.n., ,t 

IJ f.* 'tit. < I'.i’i i< ii.i.i- h' l"’ im- 
I'Min-.s.. .(.{H. t,. ,1 .M.il-lv 

• .i( V.*'* ■" np.M '*.' . Ml H'l’. 

t >' uiMM'it J.i Ml I i't>. I-'.' .inly n'(.if nj ) 

'I., l .|1 f If I <! I ..I -K T , 1 . V 



TECHNIQUES FOR IMPROVING THE DIAGNOSTIC PROCESS 



School psychologists spend between 40%-60% of their time in 
assessment activities (Anderson, Cancelli, & Kratochwill, 1984; 
Gutkin £. Conoley, 1990). However, there is abundant research 
which indicates that assessment practices vary between states, 
districts, and individual practitioners (McDermott, 1980). 

The school psychology literature encourages expanded roles 
for school psychologists, reduced emphasis on training in 
assessment, and more training in consultation skills (Fagan, 

1986, 1989; Grimes, 1989). However, in the majority of school 

districts, assessment continues to consume a significant amount 
of time for school psychologists. The Whelan and Carlson (1986) 
review of books in school psychology from 1970 to 1986 led to the 
observation, "Rather than assuming diminishing importance, as 
anticipated by the literature of the early 1970' s, assessment 
functions have broaden in scope and sophistication" (p.286). 

This paper will address methods to improve the diagnostic 
process. It is important to remember that under examination in 
this paper is the diagnostic process . This means that from the 
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time of referral to the time of the eligibility conference 
techniques can be employed to improve that process. 

Some techniques that will be introduced include: 1. 
Statistical or Actuarial Methods of Assessment; 2. Multitrait- 
Multimethod Approach to Multif actored Assessment (MTMM) ; and 
Direct Instruction in Reasoning and Decision Making. In 
addition, discussion about the changing role of the School 
Psychologist in relation to assessment will be examined. 

STATISTICAL OR ACTUARIAL METHODS OF ASSESSMENT 

The statistical or actuarial method of assessment is a 
manner of combining data in an automatic or routinized manner 
based upon previously established, empirically derived relations. 
Actuarial methods have two major advantages. First, actuarial 
procedures always lead to the same conclusion for a given data 
set. Second, the mathematical features of actuarial methods 
ensure that variables contribute to conclusions based on their 
actual predictive powers and relation to the criterion of 
interest . 

Achenbach & McConaughy (1987) listed psychometric principles 
as they apply to assessment. These principles were summarized as 
follows : 

1. Assessment should employ standardized procedures. 
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2. Multiple items should be used to sample each aspect of 
functioning . 

3. Items should be aggregated to provide quantitative 
scores for each aspect of functioning. 

4. Scores should be normed to indicate how an individual 
compares with relevant reference groups. 

5. For variables potentially related to development, the 
normative reference group should be formed according to 
age levels or other indices of development. 

6. To be considered psychometr ical ly sound, assessment 

procedures must be reliable and valid, although the 
types of reliability and validity vary with the type of 
procedure (p. 13). 

Dawes (1988) reported: "the finding that linear combination 

is superior to global judgment is strong: it has been replicated 

in diverse contexts, and no exception has been discovered" 
(p.209). The actuarial method has time and again proven superior 
to expert clinical judgment (Dawes, 1988) . Even improper linear 
models have been found to be superior to clinical judgment 
(Dawes, Faust, & Meehl, 1989; Piacentini, Cohen, & Cohen, 1992). 
Although the actuarial method of assessment may be superior to 
the clinical method of assessment, it too, achieves only modest 
results, Macmann, Barnett, Lombard, Bel ton-Kocher , and Sharpe 
(1989) wrote: "it does not appear that the classification 
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problem can be resolved through better statistical formulas" 
(p.l43) . Later they added: "Quite simply there doss not appear 
to be an objective basis for (Siagnostic classification" (p. 145). 

In spite of this finding, many continue to argue that there 
is a place for statistical applications in psychoeducat ional 
evaluations (Dawes, 1988; Dawes, Faust, & Meehl, 1989; McDermott, 
1980, 1981, 1986). At the very least, clinical and statistical 

methods can be integrated in a way that draws upon the strengths 
of each methodology and counteracts the weaknesses of each 
(Sicoly, 1989). Several psychoeducational assessment instruments 
have recently been developed using actuarial principles. These 
include the revision of the MMPI and the Achenbach scales. These 
instruments are the first steps in the emergence of the actuarial 
method of test construction. 

Several software programs based upon statistical methods 
have been developed. The McDermott Multidimensional Assessment 
of Children (M-MAC) , (McDermott & Watkins, 1985) is an example of 
a computerized, statistically based diagnostic decision aid. 

This instrument should not be seen as the final word on any 
diagnostic question, but it can be used as an aid in making 
diagnostic decisions. In particular, it may assist in overcoming 
the problem associated with the unreliable manner of combining 
data from different sources to make diagnostic decisions. The M- 
MAC reliably combines test information from a variety of 



O 

ERIC 



Davidow-5 



settings, which increases reliability and, when interpreted 
properly, should increase the validity of the decision making 
process . 

Glutting (1986) wrote: "Perhaps M-MAC's greatest 

contribution to child exceptionality is its reliability. 

Research had demonstrated repeatedly that classifications tend to 
be unreliable, both across and within specialists" (p. 334). 

Unfortunately, a revision of the M-MAC to include updated 
assessment instruments (i.e. WISC-III) was cancelled because of 
technical difficulties with the work (APA, 1994) . 

MIJLTITRAIT MULTIMETHOD APPROACH TO MULTI FACTORED ASSESSMENT 

Gresham (1983) has suggested the use of a multitrait- 
multimethod (MTMM) approach to assessment as a means of 
overcoming assessment problems. This method is based on Campbell 
and Fiske's (1959) multitrait-multimethod means of establishing 
the construct validity of psychological and educational tests. 

Scoring from measuring instruments are often affected by 
variables other than the trait being assessed by the instrument. 
Sometimes scores are more influenced by the person administering 
the instrument or the type of assessment device being used than 
by the trait being assessed. For example, when being interviewed 
about their interests, students may respond in one manner to 
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their parents, in another manner to their guidance counselor, and 
in still a different manner to their friends. In such an 
instance, the results of the interview may be a function of the 
"interviewer" rather than of the trait being assessed by the 
interview. Campbell and Fiske (1959) use the terms "trait" 
variance and "method" variance to differentiate between the 
extent to which scores on a measuring instrument are influenced 
by: a. the trait being assessed and b. the method used to assess 
the trait. Clearly, it is desirable to have trait variance 
exceed method variance. That is, the validity of the scores 
yielded by various assessment techniques can be evaluated by 
using multiple techniques (methods) to assess multiple traits and 
then evaluating the extent to which these various scores agree 
with one another. Validity exists when there are higher 
correlations between different methods of assessing the same 
trait than there are between the same methods of assessing 
different traits. In summary, different methods of measuring the 
same trait should converge on that trait and account for more 
variance in measurement than the same method of measuring 
different traits (Gresham, 1983) . 

The MTMM approach to assessment can be applied by school 
psychologists' to "assess" the validity of the assessment 
techniques being utilized and to improve the reliability and 
validity of the overall assessment process. In particular, the 
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MTMM approach can be used to overcome the problem of technically 
inadequate tests. 

Consider the following example cited by Levinson (1993): 
Assume that an interest inventory that possesses less than 
desirable reliability and validity was used in an assessment, and 
that the results indicated high interest in automotive mechanics. 
Although it would be inadvisable to place much confidence in the 
interest inventory results alone, assume that the student had 
also indicated high interest in automobile mechanics in an 
interviev/ with a guidance counselor and had been observed 
rebuilding car engines by parents and teachers. That multiple 
methods of assessing interests (tests, interviews, observations) 
all yielded similar results suggests that the interest inventory 
results, in this instance, may be reliable and valid. Had there 
not been agreement among the results of these different 
assessment techniques, the validity of the assessments would be 
in question, and considerably less confidence could be placed in 
the results. 

Relatedly, the extent to which assessment data may be a 
function of the individual gathering the data (or the interaction 
between the student and the data gathered) can also be evaluated 
by ensuring that different professionals are responsible for 
gathering similar information about a student using similar 
assessment techniques. That is, if both a school psychologist 
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and a teacher were to interview a student about his or her 
interests and both were to conclude that the student was 
interested in automotive mechanics, greater confidence could be 
placed in this consultation than would be the case if only the 
school psychologist had interviewed the student. In contrast, if 
what the teacher and school psychologist concluded from 
interviewing the student differed, considerably less confidence 
could be placed in this data. In this instance, one might infer 
that the student's responses may have been influenced by the 
interviewer. The school psychologist should also look for 
corroboration by trying to determine whether the behavior occurs 
across situations or environments. Obviously, it is extremely 
risky to base psychoeducat ional decisions on one sample of 
behavior or one test performance. 



DIRECT INSTRUCTION IN REASONING AND DECISION MAKING 

The research concerning direct instruction in 
psychoeducat iona 1 decision making is scarce and the 
findings are mixed (Fagley & Asher-Schultz , 1987; lagley & 

Kruger, 1986; Shavolson et al., 1977). However, recent research 
suggests that certain types of academic training seems to have a 
significant effect on thinking about everyday life events 
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(Lehman, Lempert, and Nisbett, 1988). 

Hernstein, Nickerson, DeSanchez, and Swets (1986) and 
Nickerson, Perkins, and Smith (1985) have shov/n that strategies 
of reasoning can even be taught to elementary school children. 
Such training not only improves performance on IQ tests but also 
improves the quality of children's open ended arguments. Arkes 
(1981) suggested several possible techniques to minimize bias for 
clinical psychologists, such as considering alternative 
hypotheses for a long period of time, using probabilistic 
information like base rates, and decreasing reliance on memory by 
taking notes. 

The objective is to formulate a plan of action which 
conceptualizes the problem broadly enough to present a suitable . 
range of alternative paths toward a correct solution. 

Instructional methods which have been found effective in 
aiding problem solving are those which focus on development of 
sound mental models. One instructional method taken from the 
work of Redding (1990) is presented below: 

1. Representation can be enhanced by providing advance 

organizers (Ausubel, 1960) prior to task presentation. 
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2. Graphic representations of problems should be used 
whenever possible, in conjunction with verbal 
descriptions . 

3. Knowledge should be provided in hierarchical structures. 
Hierarchical organization is one of the most pc./erful 
heruistics for organization information. The human 
information-processing system is organized and stored 
hierarchically in memory (Redding, 1990) . Information 
is organized in terms of subordinate, basic-level, and 
superordinate concepts (Rosch, Mervis, Gray, Johnson, & 
Boyes-Braen, 1976) . Children learn basic-level 
concepts first and with greatest ease (Mervis & Crisafi, 
IS 32). It may follow that adults learn new information 
when it is presented at some "optimal level" of 
organization . 

4. Problem-solving training should be organized around 
"problem types" to facilitate memory organization 
(DeJong & Hessler, 1986). 

5. Subgoals and the procedure for reaching them should be 
provided . 

6. In sequencing instruction of problem solving steps, the 
instructor should guide the student by shifting to new 
problem states or subgoals rather than regressing to 
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7. Instruction by way of analogy to similar problems should 
be encouraged, but the analogies must be simple and 
explicit (Reed, Ernst, & Banerji, 1974). 

8. Flexibility in problem solving should be taught by 
encouraging active student participation, questioning, 
and experimentation. Students should understand that 
different problem situations will require adaptability 
and modification of prior mental models and heuristics. 
Students must be encouraged to practice their skills and 
strategies in multiple contexts and ranges of 
complexity . 



Although more research is needed, particularly in regard to 
the effect of direct instruction in reasoning and decision making 
for school psychologists, the incorporation of direct instruction 
in school psychology training programs may prove helpful. 



OTHER STRATEGIES 

To reduce problems associated v/ith the use of inconsistent 
decision making rules and varying theoretical orientations. 
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McDermott (1981) urged training programs to use techniques like 
Thorne's (1970) "Psychological Twenty Questions", which is like a 
game of twenty questions where students learn how to gather 
information effectively. The "Psychological Twenty Questions" is 
a formal device for the teaching and objective measurement of 
psychological interviewing. A selected set of 20 protocols was 
constructed giving instructions for one person to play the part 
of the client. The student interviev.'er is allowed 20 series of 
questions or a 50 minute interviev/ in which to elicit significant 
cues and facts as given in the protocol of the case. Two methods 
of scoring are available involving either the gross number of 
lines of questioning needed to elicit the required facts, or the 
absolute number of facts elicited during the 50 minute interview. 

Engin and Miller's (1975) multimedia simulation, titled 
"PSYCHISM", places students in simulated real-world situations 
and asked them to make the best possible decisions. Simulation 
allows participants to "try out" possible behavioral alternatives 
in a relatively non-threatening situation wnere errors can be 
examined and behavior modified. Simulation can be viewed as one 
way of executing a more carefully planned transition from 
didactic instruction to the internship experience in school 
psychology. "PSYCHISM" is based on the supposition that enacted 
response experiences will best prepare the participant to respond 
to the actual situation. Optimal transfer of learning should 
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occur when both the stimulus and the response situation of 
instruction closely approximate the real situation. 

McDermott (1981) also noted that the programmed case study, 
which is a hybrid of programmed instruction and the case study, 
has been shown to maximize the diagnostic accuracy of both 
nomothetically and idiographically oriented diagnosticians 
(Dailey, 1971; Fancher, 1966). Standardization in the evaluation 
procedure has also manifested itself in diagnostic models in 
psychiatry (Spitzer & Endicott, 1974; Spitzer, Endicott, Cohen, & 
Fleiss, 1974) and clinical psychology (Nathan, 1967). McDermott 
and Watkin's (1985) multidimensional actuarial classification 
system (M-MAC) is a similar attempt at standardization in school 
psychology. Use of these techniques may help school 
psychologists become more sensitive to the problems encountered 
when trying to gather the information necessary to make a 
diagnosis, and might help standardize the method that school 
psychologists use when trying to obtain information. 



THE SCHOOL PSYCHOLOGIST'S RELATIONSHIP TO ASSESSMENT 

The task of assessment for the school psychologist is an 
arduous one. Where once the stock and trade of the psychologist 
in the school was assessment, today there is a substantial 
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lack of support for that role in the literature. It is also 
evident that more focus in training programs needs to be on 
intervention and consultation, but the problems inherent in 
assessment will not disappear if v;e ignore them. Certainly, even 
before one makes a behavioral intervention, an assessment of the 
situation and the environment needs to be made. 

Training in assessment and, equally important, the 
interventions that spring forth from the assessment, need further 
scrutiny. It is not correct to say school psychological 
assessment is invalid and must be discontinued. It is more 
correct say that there are areas where psychological and 
psychiatric assessment is flav/ed. V'/e must work to improve in 
those areas. 

It is my contention that psychological evaluations will be 
read if they contain information that teachers, parents, and 
other professionals can use. It is also my contention that when 
equivocal judgments are made by school psychologists, they need 
to be plainly stated in the report. 

A proper assessment helps "frame" the possible interventions 
that can be made. An assessment should not be looked at as 
etched in stone, but rather as a snapshot in the moving current 
of a child's life. Change, v/hicli I believe to be the raison 
d'etre for therapy and interventions of any sort, should also be 
reflected in the reports we write. Cunt lad ict ions in measures 



‘ BEST COPY AVAILABLE 



Davidow-15 



may not be indictments of the psychological tools we use 

but accurate reflections of human personality. Let's not forget 

the words of Walt Whitman who writes: 

"Do I contradict myself? 

Very well then ... I contradict myself; 

I am large ... I contain multitudes". 

Human functioning is much like what Whitman writes about in his 
poetry. Human existence is a highly complex, complicated affair 
replete with contradictions. School psychologists must be 
acquainted with the ebb and flow and ever changing tides of life. 
Our students live their lives in three dimensions and our reports 
must begin to reflect the subjectivity that involves any human 
interaction . 

The focus on intervention that many recent theorists so 
eloquently write about carries v/ithin it the implicit assumption 
of assessment. As previously mentioned, in any problem-solving 
model there needs to be an assessment phase to determine what, if 
any, intervention can be implemented. 

In addition, when working on assessments it's important not 
to lose sight of the purpose of the evaluation. What will 
classification of this student afford him or her in terms of 
services. The key to any assessment is the amelioration of 
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difficulties. It may be that school psychologists' can serve 
their school communities more efficiently by providing not only 
traditional assessments but also by providing support and 
consultation to everyone involved in the school community. 

Changes are taking place in the school environment and 
school psychologists' must expand their roles. However, the key 
to keeping the psychologist in the school is by adding more 
behavioral flexibility and training so school psychologists' 
remain a critical resource in the school culture. 



SUMMARY 



This paper addressed methods to improve the diagnostic 
process. Techniques mentioned to improve the process included: 
the actuarial method, the multitrait multimethod approach to 
mult if actored assessment, and direct instruction in reasoning and 
decision making. In addition, the changing role of the school 
psychologist in relation to assessment v;as discussed. 

The paper also discussed the role traditional assessment 
still has in a school psychology practice. It was suggested that 
there is a need to refine and improve that process rather than 
discard it. It was also suggested that the key to keeping school 
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psychologists' in the school setting is by expanding their 
training to provide more varied services to the entire school 
community . 
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